[TECH-278] Cron, Drush and per-job monitoring with an admin overview - #6

Adds cron and Drush health endpoints, an admin overview page, and per-job Ultimate Cron detection — and closes the gap the logs endpoint cannot see: a check dying silently, with no error keyword written anywhere.

Supersedes #5. This branch contains those two commits plus five more, so #5 can be closed in favour of this one.

Why

/admin/reports/monitoring/logs matches error text. That catches a noisy failure and nothing else — if cron is disabled or Drush stops working, nothing is written, the old lines stay clean, and the endpoint reports 200 forever. The new checks answer "when did this last succeed?" instead, which is a question silence can fail.

New features

Cron endpoint — /admin/reports/monitoring/cron. Compares system.cron_last against a configurable max age. Core sets that timestamp only after handlers and queues finish, so a run killed part way through never advances it.

Drush heartbeat — /admin/reports/monitoring/drush, plus a drush logs-monitoring:heartbeat command and an optional wrapper script. A web request cannot test CLI Drush, so the check is inverted: the CLI bootstraps Drupal, runs a real database query and an entity query, and only then records a timestamp. The write is last on purpose — any failure above it leaves the old timestamp and the endpoint goes stale. A dead man's switch, not a report that can itself go missing.

Overview page — /admin/reports/monitoring shows the live result of all three checks, the per-log breakdown, failing cron jobs and the CLI diagnostics from the last heartbeat. Answers "is the alert right?" without waiting for a poll. Lists each endpoint as an absolute link ready to paste into a monitor, and has a Settings tab.

Status report rows — all three checks appear on /admin/reports/status, including the one thing an endpoint cannot express: a heartbeat that was never recorded because Drush 12.5 is missing from the codebase looks identical to one that is broken.

Ultimate Cron submodule — logs_monitoring_ultimate_cron. Ultimate Cron runs each job independently and swallows per-job failures by design, so one broken job leaves system.cron_last looking perfectly healthy. The submodule inspects every enabled job and reports degraded naming the offenders, using each job's own schedule to judge lateness. It plugs in through two documented hooks (logs_monitoring.api.php), so the main module contains no reference to Ultimate Cron at all.

Improvements to the existing logs endpoint

• A missing log no longer erases a real error. The status code was assigned inside the loop, so a later healthy — or missing — log reset an earlier failure back to 200. One rotated log could hide a genuine error in another.
• Unreadable logs now fail. A log that cannot be searched is not evidence of health.
• Per-log staleness. Uses the filemtime() the endpoint already read but never decided anything with, so a scheduled writer that stops running is caught even though it writes no error text.
• Shared base class for the cron and Drush checks, so the two endpoints cannot drift apart in status vocabulary or cache headers.
• Supported versions cut to ^10.3 || ^11; Drupal 8/9 are EOL and the promoted properties and Drush attributes here never worked on them.
• CI running PHPCS, PHPStan, PHPUnit and shellcheck against Drupal 10.3 and 11.

Behaviour changes to be aware of

• The logs endpoint returns 503, not 500. Both are non-2xx, so existing UptimeRobot monitors keep working; 503 keeps a health verdict out of the error tracking a 500 lands in.
• max_age is required per log, defaulting to an hour. For a log only written when something breaks, where silence is healthy, set a long value (31536000) to keep the error-word search without the staleness check.
• Endpoints moved under /admin/reports/monitoring/. The old /admin/reports/logs-monitoring still works and returns the identical response, since it is already configured in monitors. Deliberately not a redirect: whether a 3xx is followed is the monitor's choice, and an unfollowed one reads as up.

Deploy step — please don't skip

The update hooks create rest.resource.logs_monitoring_{cron,drush} and grant role permissions. On a config-managed site the next drush config:import deletes anything absent from the sync directory, which silently removes the endpoints and leaves the monitor alerting with no visible cause.

After updb, run drush config:export and commit the two rest.resource.* files together with user.role.anonymous and user.role.authenticated.

Fixes found by running this on a real site

• hook_requirements would have crashed /admin/reports/status on Drupal 11.3+. The REQUIREMENT_* constants were replaced by a RequirementSeverity enum in 11.3 and removed in 11.4, so naming them raises "Undefined constant" — fatal under PHP 8. Now resolved at runtime, so 10.3 and 11.4 both work.
• Undefined property: DatabaseLogger::$name. Ultimate Cron's own CronJob::loadLatestLogEntries() reads a property the Drupal 8+ logger plugins do not have. A warning is not a \Throwable, so a try/catch cannot make it safe — it fills the log and returns a wrong answer. Jobs are now grouped by logger and dispatched to the working instance method.
• sh drush-healthcheck.sh reported Bad substitution. Running it with sh bypasses the shebang; it now re-execs under bash rather than half-running.
• A relative --drush path failed with timeout: no such file. The binary was validated against the caller's directory, then the script cds into the Drupal root. It is now resolved to absolute first, and the failure message names the binary and the directory.

Testing

Run against a Drupal 11.4 site with the module and Ultimate Cron installed:

Kernel:     49 tests, 122 assertions — pass
Functional: 17 tests,  87 assertions — pass
PHPCS:      Drupal,DrupalPractice — clean

Coverage targets the contract that matters rather than the happy path: each check returns the right code for ok / stale / never / degraded, a missing log cannot mask an error found in another, the verdict is never served from a cache, host diagnostics are withheld from anonymous, the superseded logs path returns byte-identical output, and the Ultimate Cron inspection raises no PHP warning.

Steps to test

• /admin/reports/monitoring/cron returns 200, then drush sset system.cron_last 1 → 503 stale
• drush state:delete logs_monitoring.drush_last → /admin/reports/monitoring/drush returns 503 never; drush logs-monitoring:heartbeat → 200 ok
• All three endpoints reachable anonymously; /admin/reports/monitoring is 403 anonymously
• /admin/reports/logs-monitoring still answers
• /admin/reports/status lists the cron, Drush and log rows
• With logs_monitoring_ultimate_cron enabled, a failing job shows on the overview page and turns the cron endpoint 503

Known deprecations, left deliberately

@RestResource annotations and the procedural hook_requirements both warn on Drupal 11.3+ and are removed in 13. Their replacements do not exist on the ^10.3 floor, so adopting them now would break the declared range. Note that #[LegacyRequirementsHook] is not the fix — it suppresses the legacy hook, and without an OOP replacement it silently empties the status report.

Not covered

The wrapper script's multisite path is exercised only by hand; there is no automated test spanning several sites.